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ABSTRACT 

"virtual" institution that offers baccalaureate and master’s degrees in 
business communication via classes conducted entirely on the World Wide Web 
and Internet. Courses are developed by experts in the field of business 
communications; IU then compiles the necessary study guides, reading packets, 
interactive computer software, Web sites, and videos. In 1996, in preparation 
for the accreditation process by the North Central Association of Colleges 
and Schools, IU contracted with the Flashlight Project for the design of a 
comprehensive institutional evaluation plan which focused on four issues: 
academic content, academic resources, retention, and technological 
opportunities and challenges. This paper examines how the virtual environment 
affected the evaluation design and process, and what the evaluation revealed 
about IU's ability to reach its learning goals of fostering collaboration, 
encouraging student- centered learning, and enabling productive interaction 
between students and faculty and among students. The paper discusses how a 
virtual university is evaluated; the IU student experience; and goals for 
student interaction with faculty and other students, for student professional 
enrichment, and for student team work. The paper concludes that the 
evaluation has been successful (accreditation was granted in 1997) , and also 
led to a process of continual quality improvement. (CH) 
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In 1995 approximately 5 percent (750,000) of higher education students were enrolled in distance 
learning programs. At that time most institutions were offering distance learning courses by way of “two- 
way interactive video” (57 percent) or “one-way pre-recorded video” (52 percent). Only 14 percent 
offered courses with “two-way online interactions” (the Internet) and the World Wide Web wasn’t even a 
separate category (U.S. Department of Education, 1995). These were the first real data available on 
distance learning, but they are already outdated. In just three short years it is safe to assume that these 
numbers have changed significantly. The number of students accessing education electronically and the 
percent who study via the Web are both likely to be larger today than in 1995. Since the mid-1990s, 
distance learning and use of the WWW, in particular, have surged. According to K.C. Greene’s Campus 
Computing Survey “Internet Resources” which were used by barely 1 percent of courses in 1994 were used 
by 25 percent of courses in 1997, while the World Wide Web which also barely registered in 1994 was 
used by nearly 15 percent of courses in 1997 (Green, 1997). 

Not everyone in the academy is embracing the use of these new technologies. Some scholars 
question the value of all types of education delivered electronically, concluding that by definition this is a 
third rate form of education (see for example Ashworth, 1996; Cordes, 1998). It is becoming increasingly 
important to have timely and reliable information on how information technologies are being used in 
instruction and what effect they are having on the teaching and learning process and student outcomes. 

This paper discusses an evaluation that was designed to help a “virtual” institution answer these questions. 

International University is an independent, nonprofit “virtual university” that offers a Bachelor of 
Arts and Master of Arts degree in business communication. All classes are conducted entirely by way of 
the Intemet/World Wide Web. International University, “The University of the Web,” was founded by 
Jones International, LTD., in 1995. All International University courses are developed by leading experts 
in the field of business communication. International University then compiles the information necessary 
for teaching each course in study guides, reading packets, interactive computer software, World Wide Web 
sites and videos. In 1997 International University was awarded candidacy for accreditation by the North 
Central Association of Colleges and Schools (NCA). 

In 1996, in preparation for the accreditation process and in order to begin an internal process of 
continual quality improvement, International University contracted with the Flashlight Project to develop 
and begin implementation of a comprehensive institutional evaluation. The Flashlight Project was selected 
to design the evaluation because of its emphasis on how the use of specific information technologies 
affects the teaching and learning process. The Flashlight evaluation process begins with the assumption 
that the technology itself does not change teaching or learning. It is the way in which technologies are 
used to enable and/or enhance particular teaching and learning strategies that makes the difference. If an 
institution or program simply purchases computers, but does not change the way in which the courses are 
taught via the computers no real advantages or disadvantages that may result from having computers in the 
classroom will be realized. At their best, technologies are empowering. That is they present new options 
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for teaching and learning that were not apparent before their introduction. The goal of the Flashlight 
project is to help institutions discover how their students and faculty are using the technologies available to 
them and uncover the ways that enhance the learning process and ways that hinder it (see for example, 
Ehrmann, 1995; Chickering and Ehrmann, 1996; Ehrmann and Zufiiga, 1997). 

Evaluation Plan 

International University’s (IU) youth, small size, mature student body, and distance delivery 
mechanisms presented unique opportunities for evaluation. First, as a young institution, in only its second 
year of operation at the time the evaluation was designed, IU had the opportunity to put in place a 
comprehensive self-study process that will help it maintain a process of continuous quality improvement. 
Second, the relatively small size of the university - 180 students (30 to 40 active students per term), 1 1 
content experts and 8 teaching faculty - and IU's philosophy of student-directed learning made involvement 
of students and faculty in the evaluation process feasible and desirable. Third, IU’s reliance on the World 
Wide Web and the Internet required and enabled innovative evaluation techniques and data collection 
procedures. Finally, International University has very well defined learning goals for their courses 
(fostering collaboration, encouraging student-centered learning, enabling productive interaction between 
students and faculty and among students), which became the primary criteria for the evaluation. 

The Flashlight Project recommended that International University focus on four issues: 

Academic content : is IU providing students with courses and programs that are academically 
rigorous and appropriate to students’ educational and professional goals? 

Academic resources : is IU providing students with adequate access to a full range of academic 
resources (access to other university libraries and Internet access to resources), advising and counseling 
services? 



Retention: is IU able to retain a diverse range of students in its courses and its degree programs? 

Technological Opportunities and Challenges : what opportunities and challenges does reliance on 
the World Wide Web and other technologies (e.g., video tapes, E-mail) present for students and faculty? 

The overall evaluation plan included methods for collecting data for answering each of these four 
questions. The methods developed include analysis of student record data, questionnaires and focus groups 
of students, faculty, course and program drop-outs, graduates, and students’ employers. The evaluation 
proposed by the Flashlight project focused on the degree to which the technological tools used by 
International University (Intemet/WWW, videotapes, etc.) were enabling or hindering students and faculty 
from meeting their educational goals. 
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It is not possible to discuss all of the methods or results from the evaluation here. Therefore, this 
paper focuses on two issues: (1) how the “virtual” environment affected the evaluation design and process; 
and ( 2) what the evaluation revealed about IU’s ability to reach its learning goals of fostering 
collaboration, encouraging student-centered learning, and enabling productive interaction between students 
and faculty and among students.. Thus far, results have been completed on student retention, and faculty 
experiences, as well as the experiences of current students. However, there is not enough space in this 
paper to review all of the pieces of the study. Therefore, the following discussion is limited to how these 
issues were addressed in the collection of information from current IU students 

Evaluating a Virtual University 

In developing tools for evaluating the use of information technologies in instruction the question 
of how the evaluation itself is affected by the presence of technology in the institution has intrigued us. It 
is clear that basic research methods do not change. The researcher still must establish criteria to be 
measured and identify the primary clientele from which data must be gathered. The real difference 
between an evaluation of a campus-based and virtual institution is in the type of questions that must be 
asked and the way in which data are collected. 

Asking the right questions . As we began to approach the issue of how to develop tools for virtual 
and distance education programs we explored many of the existing student experience questionnaires (such 
as the College Student Experiences Questionnaire (CSEQ); and the ACT College Outcomes and Student 
Opinion Surveys). Although many of the questions on these surveys are relevant to any form of education, 
regardless of how it is delivered, others are focused on a campus-bound experienced that is foreign to 
students in a virtual environment. The ACT Student Opinion Survey, for example, asks students to indicate 
their satisfaction with the “out-of-class availability of your instructor” and the “general condition of 
buildings and grounds”, neither of which have any relevance to a student attending from a distance. 
Likewise there are no questions in these instruments about access to the Internet or ability to connect to a 
World Wide Web page. 

In addition, the focus of these instruments is on the campus experience. In contrast we wanted to 
focus on how technology affects teaching and learning. The Flashlight evaluation strategy utilizes the 
“Seven Principles of Good Practice in Undergraduate Education” (see Table 1) developed by Arthur 
Chickering and Zelda Gamson (1987) to develop specific indicators. For example, the Flashlight “tool kit” 
includes specific indicators for measuring the extent to which electronic mail enables interaction between 
the faculty and students (faculty-student interaction); and the degree to which access to video tapes 
empowers students to spend more time studying and reviewing materials (time on task). These tools were 
used to customize instruments for collecting data from International University students and faculty. 
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Table 1 

Seven Principles for Good Practice in Undergraduate Education 

1. Encourages Student- Faculty Contact 

2. Encourages Cooperation (Collaboration) Among Students 

3. Encourages Active Learning 

4. Prompt Feedback 

5. Emphasizes Time on Task 

6. Communicates High Expectations 

7. Respects Diverse Talents and Ways of Learning 
Chickering and Gamson, 1987 

Data Collection . The way in which we were able to collect data from IU students were both 
constrained and expanded by the virtual environment. First, Distributing end-of-course student evaluations 
during a class period, while practical on many campuses, was not a possibility in the virtual environment. 
Students at IU work at their own time and pace. The use of the World Wide Web as a medium for 
distributing the courses, however, gave us the opportunity to use the Web for disseminating and collecting 
questionnaires. In a campus-based environment, Web-based questionnaires cannot stand alone because 
not all students are technologically sophisticated. In this situation relying solely on web dissemination and 
collection can introduce bias and reduce the response rate. In a ‘virtual’ environment, however, use of the 
web for data collection is very viable. All of the students must be proficient at use of the Web in order to 
take the courses. Therefore, the technology itself does not introduce any bias. 

Students were asked to submit their student identification number. E-mail addresses are 
automatically submitted with the completed questionnaire, but are not a good unique identifier of student 
responses. However, they can be used to check for duplicate responses. 

The web-based questionnaires were successful. Pilot tests were conducted and errors in 
programming the questionnaires were corrected before initiating the full evaluation. Response rates for the 
three terms (IU operates on the basis of six eight-week course modules/year) that have been analyzed by 
the Flashlight project staff were very high (see Table 2). Moreover, The data were submitted electronically 
and automatically loaded into a database for analysis and the resulting database was very clean requiring 
little, if any, auditing. 



1997 Sessions 


Table 2 


N 


% 


Summer II 




27 


84.4 


Fall I 




24 


70.6 


Fall II 




19 


70.4 
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The ‘virtual’ environment also presented challenges for conducting focus groups of students. IU 
students are scattered across the nation and throughout the world. It is impossible to bring students together 
for a face-to-face meeting. A telephone focus group was considered as one option. Since students also are 
scattered across numerous time zones this also was not a viable option. Another option was using 
International University’s electronic “chat” software to conduct a “real-time” electronic focus group. This 
option was abandoned, however, due to technological problems with the “chat” software IU was using at 
the time. Since all students must use the Internet for electronic mail discussions during their courses a new 
opportunity presented itself. We decided to conduct the student focus group by way of an asynchronous, 
electronic-mail session. 

The student focus group was conducted over a period of ten days. A sample of students from the 
Fall I term were asked by electronic mail to participate. A special discussion list was established for the 
focus group, and each participant was notified when they were attached to the list. Shortly before the 
beginning of the focus group each participant was asked to respond to the discussion list by introducing 
themselves to the group and telling us something about themselves including the courses they had taken at 
IU. A moderator guided the discussion. One question was distributed to the group for comment once 
every day or two, depending on the type of question. Everyone was able to read each others comments and 
participants were encouraged to respond to each other. 

The single biggest problem with electronic mail focus groups is keeping participants on task. 
Because the focus group is not “focused” at a particular time and place, the moderator must continually 
encourage participants to respond in a timely fashion. The focus group was originally scheduled to last 
only 5 days. As the discussion continued, however, it became clear that more time was needed for 
participants to respond to questions and the moderator made a decision to extend the discussion time. This 
was important if we wanted to capture detailed information from the students. 

As with telephone focus groups, body language signals are lost in an electronic mail focus group. 
However, similar to telephone focus groups respondents in electronic mail focus groups appear to be more 
candid with their responses (Silverman, 1997) and, characteristic of much of the anecdotal information 
about electronic mail use by students, their comments tend to be more thoughtful. Many students provided 
long detailed stories about how their experience at IU had affected them personally and professionally. 

Overall the electronic mail focus groups were successful. Nevertheless, electronic mail focus 
groups are very new and more systematic research needs to be conducted on how the electronic mail 
environment affects responses and the role of the moderator. 

The IU Student Experience 

International University caters to a very specific student clientele - mature, working professionals. 
It should not be surprising then that the youngest IU student in this study was 23 years old, or that nearly 
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all of these students indicated they are employed and that most arrive at IU after completing at least a 
bachelor’s degree. 

The goals IU has for its courses are targeted to the needs of this clientele. First, they want to 
ensure that students have adequate interaction with faculty and other students. Second, they have 
developed a program that they hope is directly applicable to the professional development needs of their 
students. Finally, because working in teams is important in the corporate world IU wants to make sure 
their students are given practice at working on projects in teams. The following discussion reviews the 
information received from IU students for the Summer II, Fall I , and Fall II sessions in 1997. 

Goals for Students: Interaction with Faculty and other Students 

The Intemet/World Wide Web and electronic communication are the primary mediums through 
which International University courses are delivered. Therefore, student perceptions about the 
effectiveness of these information technologies and their ability to foster productive interaction is 
important to the success of the overall program. 

All but one respondent in all three sessions agreed with the statement “I would recommend that 
others take a course that: uses electronic communication, such as electronic mail or computer conferencing. 
Similarly, all but one student in all three sessions said s/he would recommend that others take a course that 
uses the Intemet/World Wide Web. Moreover, many of the comments to the open-ended questions 
expressed the belief that using electronic communication added value to the educational experience. 

The percent of students who say they used e-mail for communicating with their instructor and 
other students 3 or more hours per week ranged from 35 percent in the Summer II session to 52 percent in 
Fall I and almost 78 percent in Fall II. Similarly, 29 percent of the students in the Summer II and Fall II 
sessions said they spent 1 hour or more per week participating in electronic “chat sessions,” compared to 
53 percent in Fall I. 

There are two ways of interpreting these figures. First, it is clear that electronic mail and “chat” 
sessions are not used as much during some sessions. This may be due to the relative mix of courses offered 
each session. Another way to look at these figures is as an indicator of how much students interact with 
each other and the instructor. A 3 credit hour course generally includes 3 hours of face-to-face interaction 
with the instructor and other students (1 to 2 hours of lecture and 1 to 2 hours of discussion time) per week. 
In comparison, most International University students say they are spending 4 to 6 hours per week 
interacting with the instructor and other students for the same amount of course credit. 

Still more student hours were employed in using the Internet as a medium for study. More than 
60 percent of the students in all three sessions said they spent 3 hours or more searching the Intemet/World 
Wide Web to access reference materials or conduct research. Further, more than 60 percent of the Fall I 
and Fall II students, and approximately one-third (36 percent) of the Summer II students said they spent at 
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least 3 hours per week accessing the Intemet/World Wide Web to view or download course related 
materials. A third or more of the Fall I and Fall II students said they spent more than 6 hours per week in 
this activity. This suggests that a large percentage of International University students are spending at 
least 2 hours reviewing materials electronically for every hour they participate in discussions about the 
course materials with the instructor and other students. 

Goals for Students: Professional Enrichment 

International University is designed to provide professional enrichment for those employed in the 
communications industry. In fact most students say they are taking a course at this time for a job related 
reason: “to advance in career” (Summer II and Fall I) or “meet an employer’s requirement” (Fall II). 
Therefore, it is important that IU courses provide instruction that is applicable to the type of problems 
students face as professionals. 

In general, IU appears to be meeting this goal. The vast majority of students (60 to 100 percent), 
in all three sessions, say they agree with the statements: “Assignments for this course are similar to the 
type of tasks I face as a professional;” and “I can apply what I have learned in this course to my job.” 
However, there were some negative comments by students in a course on using the Internet in business, 
about the use of dated texts and course materials. Because these materials were out-of-date some students 
complained the applicability of this course to their professional roles was limited. 

One of the unstated goals of International University courses is to help students develop 
professional networks. Therefore, we wanted to know whether or not International University students 
were discussing course related issues with professional colleagues not associated with International 
University and whether they were making professional contacts in their courses that would extend beyond 
the course itself. A majority of students say they spend at least some time discussing this course 
electronically (through electronic mail or a “chat” session) with someone who is not affiliated with IU. 
Close to 50 percent of students in each of the three sessions also says they spend at least some time 
communicating with IU instructors or other students for work or personal reasons. 

Goals for Students: Team Work 

The ability to effectively work in teams is important to a successful business career. One of IU’s 
goals is to help students develop these skills. Most students feel that the course they are taking at IU is 
helping them learn to work in teams/groups, and most also feel that working in a team helps them to 
understand the ideas and concepts being taught. 

There were, however, several students that expressed a frustration with group assignments in 
specific courses, usually because either their goals were different from the other members of their group; 




10 



7 



the group did not function well; or because the assignment was not very useful to their goals. Some 
students expressed frustration that effective team work was not possible because there were only one or 
two students enrolled in the course. Nevertheless, most of those who commented did believe that the use of 
electronic communication facilitated team work and made working in teams possible for students who must 
learn at a distance. 

Conclusion 

In general, the International University evaluation has been very successful. International 
University has begun a process of continual quality improvement. The information gathered, thus far, has 
helped International University make improvements to the way in which specific courses are conducted 
and in the way in which information is shared between content experts (who design the course) and 
instructional faculty (who facilitate the course). It also pointed out some problems with specific Web 
communication software that IU was then able to correct. 

Moreover, although there is still much to be learned about how to evaluate a “virtual university” 
we gained several insights. First, we have developed a set of instruments for measuring issues that are 
relevant to the “virtual learning” experience. The Flashlight and IU “tools” measure issues directly related 
to the “virtual” experience, such as how well specific technologies are functioning and how easily students 
are able to access needed learning resources from remote locations. Further, these “tools” focus on how 
the use of various technologies enhances or hinders learning practices which have been shown to lead to 
improved learning outcomes. We are thus better able to respond to critics who claim that distance 
education is by definition low quality education. Second, we have learned some things about administering 
surveys on the World Wide Web, and even more about the benefits and problems with conducting 
asynchronous, electronic-mail focus groups. More research needs to be conducted on how to conduct 
asynchronous, electronic focus groups and on the potential for conducting “real-time” electronic focus 
groups using “chat” software. 



11 



8 



References 



Ashworth, Kenneth H. 1996. “Virtual Universities Could Produce Only Virtual Learning.” The Chronicle 
of Higher Education, September 6, 1996, p. A56. 

Chickering, Arthur and Stephen C. Ehrmann. 1996. “Implementing the Seven Principles. Technology as 
Lever.” AAHE Bulletin , October 1996, p. 3. 

Chickering, Arthur and Zelda Gamson. 1987. “Seven Principles for Good Practice in Undergraduate 
Education.” AAHE Bulletin, March, 1987. 

Cordes, Colleen. 1998. “As Educators Rush to Embrace Technology, a Coterie of Skeptics Seeks to Be 
Heard.” The Chronicle of Higher Education , January 16, 1998, p. A25. 

Ehrmann, Stephen C. 1995. “Asking the Right Questions: What Does Research Tell Us About Technology 
and Higher Learning?” Change , March/April 1995. 

Ehrmann, Stephen C. and Robin Etter Zufliga. 1997. The Flashlight Evaluation Handbook. Washington, 
DC: American Association for Higher Education. 

Green, Kenneth C. 1997. 1997 Campus Computing Survey. Claremont, CA: Campus Computing Project. 

Silverman, George. 1997. Introduction to Telephone Focus Groups, http://www.mnav.ccom/phonefoc.htm. 
retrieved January 30, 1997. 

U.S. Department of Education, National Center for Education Statistics. 1997. Distance Education in 

Higher Education Institutions. NCES 97-062. By Laurie Lewis, Debbie Alexander, and Elizabeth 
Farris. Project Officer, Bemie Greene. Washington, DC. 




1 2 



9 



U.S. DEPARTMENT OF EDUCATION 

Ottlce of Educational Research and Improvement (OERI) 
Educational Resources Information Center (ERIC) 




NOTICE 



REPRODUCTION basts 



This document is covered by a signed “Reproduction Release 
(B anket) form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a “Specific Document” Release form. 



This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release 
form (either “Specific Document” or “Blanket’^. 




